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The  two  main  goals  of  this  project  were  to;  1 )  Describe  the  way  neurons  in  the  motor  cortex  encode  the  posture 
of  the  wrist,  hand  and  finger  during  reach-to-grasp,  and  2)  Demonstrate  the  control  of  an  anthropomorphic  arm 
and  hand  using  brain-control  in  a  task  where  a  robot  arm  reaches  out  grasps,  twists  and  pulls  a  doorknob.  We 
have  made  a  good  deal  of  progress  toward  these  goals.  The  major  advances  we  have  made  in  this  reporting 
period  include  recordings  from  behaving  primates  that  now  include  EMGs  from  arm  muscles  in  addition  to 
cortical  neural  activity  and  hand  kinematics  and  the  addition  of  a  3  DOF  wrist  to  our  robot  control  experiments. 
This  allows  the  hand  to  be  oriented  to  the  grasp  object  and  is  a  major  advance  in  prosthetic  control. 

Neurophysiological  results- 

Monkeys  were  trained  to  reach  out  and  grasp  an  array  of  objects  placed  in  their  extra-personal  space  with  a 
robot  arm.  Joint  angles  from  the  23  degrees-of-freedom  of  the  wrist  and  hand  were  precisely  measured  by 
tracking  passive  markers  on  a  glove  and  sleeve  worn  by  the  monkeys.  Single-unit  action  potentials  were 
recorded  from  5  electrodes  placed  in  the  motor  cortex.  In  a  recent  enhancement,  muscle  activity  from  13-16 
implanted  EMG  electrode  pairs  is  also  being  recorded  simultaneously  with  the  other  data. 

Database 

We  have  recorded  more  than  1600  units  from  five  hemispheres  of  three  monkeys  performing  more  than 
120,000  trials.  These  data  were  used  to  characterize  the  relation  between  neural  activity  and  hand  kinematics. 
In  summary,  the  23  DOF  used  to  describe  hand  shape  could  be  grouped  into  8  eigen  postures  which  together 
accounted  for  more  than  85%  of  the  overall  variance.  Many  of  the  single  neural  activity  had  significant 
regressions  to  the  23  DOF.  The  same  neurons  also  tended  to  have  significant  regressions  to  the  principal 
component  space,  suggesting  that  the  more  efficient  mapping  is  well  represented  in  Ml  cortical  activity. 


EMG-neural  studies  have  been  carried  in  two  monkeys  in  which  a  total  of  about  300  units  were  recorded  in 
conjunction  with  chronic  EMG  recordings  of  13  muscles  of  the  hand  and  arm  and  the  corresponding  kinematics 
from  recorded  position  data. 
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Unit  and  rectified  EMG  data  from  five  repeated  trials  to  the  same  object.  Trials  were  aligned 
(vertical  bar)  to  object  contact  which  lasted  about  1  second. 
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Spike-triggered  averaging  (STA)  is  an  important  analytical  technique  to  better  understand  the  relation  between 
motor  cortical  and  muscle  activity  (EMG).  A  snippet  of  EMG  data  (e.g.  20  ms)  is  collected  after  each  cortical 
spike  and  then  averaged  together  across  spikes.  The  resulting  averaged,  pattern  typically  has  a  post-spike 
facilitatory  profile  if  the  neuron  is  acting  as  an  EMG  driver.  The  problem  with  this  technique  is  that  the 
probability  of  any  given  cortical  spike  causing  a  change  in  EMG  is  very  small,  so  that  a  large  number  of 
samples  is  needed  and  this  is  difficult  to  do  in  our  experimental  paradigm.  For  this  reason,  we  have  developed 
a  number  of  STA  enhancements. 


STA  enhancements 
Template  matching 

One  way  to  increase  STA  efficiency  (demonstration  of  post-spike  event  with  fewer  trigger  events)  is  to  use 
template-matching  which  works  by  counting  the  number  of  times  the  post-spike  waveform  matches  a  template. 
Based  on  the  assumption  that  post-spike  effects  occur  only  rarely  in  the  target  EMG,  averaging  will  tend  to 
erode  the  post-spike  profile  in  the  absence  of  a  spike.  This  does  not  happen  with  template  methods. 


Figure  2 

Conventional  STA 


Figure  3 

Rectified  EMG 
showing  threshold 


Figure  4 

STA  of  thresholded 
snippets 


An  approach  using  template-matching,  was  developed  to  detect  the  shape  of  the  post-spike  effect  for  a  single 
neuron-muscle  pair.  A  cortical  neuron-muscle  pair  with  a  significant  facilitatory  post  spike  effect,  as  detected 
by  conventional  STA,  was  used  (Fig.  2).  These  data  were  recorded  during  a  single  digit  movement  experiment 
and  32061  Spike  triggered  snippets  were  extracted  from  the  EMG  time  series  (data  courtesy  of  Dr.  Marc 
Schieber).  These  snippets  were  thresholded  and  only  snippets  containing  EMG  values  above  a  low  threshold 
were  kept.  This  threshold  was  chosen  so  snippets  with  no  EMG  activity  or  background  noise  were  eliminated 
(threshold  example  in  Fig.  3).  The  spike  triggered  average  (STA)  of  the  remaining  5059  snippets  is  shown  in 
figure  4  and  it  can  be  noted  that  a  lot  of  the  background  noise  was  eliminated  and  the  amplitude  of  the  average 
increased. 


Figure  5 

STA  of  good  matches  (red) 
vs  STA  of  poor  matches 
(blue) 


A  template  matching  procedure  was  then  employed,  where  a  Gaussian  bump 
with  a  2  ms  width  was  used.  This  was  meant  to  detect  any  isolated  action 
potentials  following  the  cortical  spikes.  The  cross  correlation  between  every 
snippet  and  the  template  was  computed  and  the  highest  cross  correlation  value 
in  the  range  of  6-1 6ms  post  spike  was  retained.  Of  all  the  5059  snippets,  only 
555  snippets  had  good  matches  to  the  template  (the  threshold  used  was  r>0.8). 
The  STA  of  the  snippets  with  good  matches  to  the  template  is  compared  to  the 
STA  of  the  snippets  where  the  template  did  not  match  well  in  figure  5.  This  STA 
can  be  used,  for  example,  as  a  starting  value  for  the  PSE  in  the  main  model. 

The  fact  that  only  one  in  every  58  spikes  shows  a  good  match  to  the  template 
and  that  the  STA  computed  from  those  spikes  accounted  for  the  majority  of  the 
amplitude  in  the  original  STA  demonstrates  how  weak  functional  connectivity 
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can  be.  Even  for  a  monosynaptic  connection,  only  2%  of  the  spikes  produced  a  post  spike  effect.  As 
validation,  the  spike  train  and  EMG  time  series  were  split  into  segments  and  were  sampled  with  repeats.  The 
template  matching  was  then  carried  out  on  the  shuffled  data  and  the  matches  with  correlation  above  .8  were 
counted.  This  was  repeated  1000  times  and  the  maximum  number  of  matches  was  73  (mean  =  48).  The  95% 
confidence  interval  for  the  bootstrapped  data  was  at  56  matches  showing  that  the  555  matches  were  much 
beyond  that  expected  by  chance. 


Average  firing  rate  around  2980  template  occurances  between  thresholds  (red) 
and  around  1688  template  occurances  with  thresh=0.2  (blue) 


In  addition,  this  approach  was  used  to  estimate  the  firing  rate  of  the 
cortical  neuron  around  times  where  the  template  occurred  in  the  EMG. 
Instead  of  performing  the  matching  on  snippets,  the  Gaussian  bump  was 
shifted  throughout  the  whole  EMG  time  series.  A  firing  rate  histogram  of 
the  cortical  neuron  (Fig.  6)  around  the  good  matches  (blue,  r  >  .2, 
n=1688)  shows  both  a  broad  modulation  a  second  before  the  match  and 
a  sharper  increase  about  30  ms  before  the  match  decreasing  to  a  low 
rate  35  ms  later.  Weaker  matches  (red,  .1  <r<.2,  n  =  2980)  showed  only 
a  broad  modulation.  This  is  an  indirect  indication  of  the  type  of 
modulation  in  this  neuron  that  tends  to  lead  to  EMG  facilitation. 


Figure  6  Match-triggered  firing  rate 
histograms 


BCI-robot  results- 

To  develop  control  of  the  more  complex  device,  we  have  first  extended  the  tuning  model  used  for 
demonstration  of  a  brain-controlled  robotic  self-feeding  task  to  the  control  of  orientation  and  hand  shape.  Then, 
we  have  developed  a  systematic  method  for  governing  the  interaction  between  decoded  cortical  control 
commands  and  robot  movement  that  provides  a  consistent  decoding  and  training  paradigm.  We  added  terms 
for  wrist  orientation  (yaw,  pitch  and  roll),  grip  aperture,  and  hand  shape  to  the  classic  regression  formula  used 
for  3D  reaching. 

At  the  beginning  of  a  daily  experiment  we  fit  this  equation  using  observation-based  learning.  The  robot  arm  is 
moved  through  the  task  while  the  monkey  watches.  At  the  end  of  the  task,  the  monkey  is  rewarded.  The 
important  result  is  that  many  neurons  in  the  cortex  are  modulated  throughout  this  observation  and  we 
determine  each  neuron’s  preferred  direction  with  this  modulated  activity  according  to  our  calibration  model. 

Calibration  Model  -  Virtual  Directional  Tuning 
<I>i  =  /3-j  V1  +  <: 

^  "  [  ^jx  j  i^ixun  1  j 


Regression  coefficients  p  are  determined  from  the  set  v  of  observed  robot  movement  parameters.  A  directional 
tuning  model  used  previously  for  3-D  device  control  is  extended  to  describe  the  encoding  of  non-Cartesian 
controlled  variables.  Control  inputs  that  describe  hand  orientation  and  shape  are  treated  as  virtual  directions 
orthogonal  to  3-D  directional  movement. 

Once  we  fit  each  cell’s  preferred  direction.  We  switch  experimental  modes  so  that  the  recorded  signal  is  now 
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controlling  the  robot  motion.  The  control  signal  is  formed  using  the  classic  population  vector  algorithm  and  is 
represented  in  a  single  high  dimensional  vector  that  contains  the  same  dimensions  as  those  used  in  the 
calibration  model. 

Prediction  Model 


The  prediction  model  used  is  a  high  dimensional  version  of  the  Population  Vector  Algorithm.  Summation  of 
individual  unit  preferred  directions  supplies  a  control  signal  in  real  and  virtual  directions.  An  automated  system 
provides  movement  commands  to  the  robot  in  real  time. 

Collaborative  Control  for  BCI 

One  of  the  major  advances  we  have  made  in  this  last  reporting  period  is  the  formalization  of  training  procedure 
that  uses  shared-mode  control  to  provide  assistance  to  the  subject  during  BCI  training.  This  assistance  is  both 
passive  and  active. 

In  the  active  mode,  an  automated  system  provides  movement  commands  to  the  robot  in  real  time.  Control 
commands  derived  from  the  monkey  cortical  implant  are  mixed  with  signals  from  the  automated  controller.  This 
system  provides  a  substrate  for  observation-based  learning  and  enables  learning  the  control  of  individual 
degrees  of  freedom  on  an  incremental  basis  while  ensuring  task  completion. 

In  the  passive  mode,  the  monkey’s  brain  control  signal  is  attenuated  for  an  individual  DOF  if  the  control  signal 
would  move  the  limb  outside  a  “desired  command  region.”  The  amount  of  attenuation  is  adjustable  and  is 
reduced  as  the  subject  learns  to  operate  the  device  to  keep  it  in  the  proper  region. 


V=  C  ( [D,U]*  +  Ct  (D,U)*)U  +  (I-C)  D, 


PosKernel(D,  U)  =  (D,  U)*  =  I  —  [D,  U]* 

U  is  an  m-dimensional  vector  of  monkey  commands.  Columns  of  D  are  basis  vectors  of  a  “desired  command 
region”  supplied  by  an  on-line  grasp  planning  system.  D ,  is  an  m-dimensional  matrix  assumed  to  represent  a 
command  directly  toward  an  optimal  robot  state.  C  is  an  m  x  m  diagonal  matrix,  its  nonzero  elements 
representing  an  “operator  command  fraction”  for  each  degree  of  freedom.  Ct  is  an  m  x  m  diagonal  matrix, 
representing  anisotropic  compliance  in  undesired  command  directions.  V  is  the  output  command  sent  to  the 
robot. 


Our  initial  effort  focused  on  the  control  of  Cartesian  hand  location  and  pinch  gripping.  In  the  past  year  we  have 


Shaded  volume  shows  the  desired  control 
region  of  the  control  space  in  the  context  of 
the  door  knob  task 
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extended  this  control  to  wrist  orientation.  A  rhesus  monkey  was  implanted  with  dual  “Utah”  microelectrode 
arrays  in  the  Ml  arm  and  hand  areas.  Upwards  of  100  isolated  units  on  the  192  available  channels  are  used  for 
brain  control. 


Full  control  of  subsets  of  7  degrees  of 
freedom  of  the  device,  including  direction, 
orientation,  and  grasp  aperture,  has  been 
demonstrated.  The  maximum  simultaneous 
control  reached  thus  far  is  full  4-DOF 
control,  with  partial  control  over  a  5th 
dimension. 


Placement  of  two  recording  arrays  in  motor 
cortex.  The  arrow  points  to  the  central 


sulcus. 


Task  schematic-  Signals  recorded  from  two  arrays  are  processed  by  an  extraction  module  to  generate  the 
monkey’s  control  signal.  During  training,  this  is  mixed  with  an  idealized  control  space  generated  by  an 
automatic  controller  in  the  grasp  arbitrator.  The  mixed  signal  is  sent  to  the  prosthetic  arm/hand  controller. 
The  behavioral  task  is  controlled  by  an  executive  module.  All  modules  run  simultaneously  in  our  custom 
software  environment. 
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Sequence  of  events  during  a 
brain-control  task  requiring  5 
control  degrees  of  freedom  from  the 
monkey-computer  system.  The  task 
shown  here  was  to  reach  for  the 
doorknob,  rotate  the  hand  to  match 
knob  orientation,  and  perform  a 
grasp  of  the  object. 


Individual  components  (transport,  orientation,  hand  shaping  and  grasping)  of  the  task  have  been  demonstrated 
separately.  We  are  now  working  to  carry  out  the  entire  task  sequentially  with  a  single  control  scheme. 


